Skip to content

feat(vm): conformant goto/label on both engines - #364

Merged
davydog187 merged 4 commits into
mainfrom
vm/dispatcher-goto-label
Jun 15, 2026
Merged

davydog187 merged 4 commits into
mainfrom
vm/dispatcher-goto-label

Conversation

@davydog187

Copy link
Copy Markdown
Contributor

Context

Second PR in the effort to drive the bytecode Dispatcher to full opcode coverage (follows #362). The remaining real gap was :goto / :label.

While implementing the dispatcher port, I found the premise was wrong: the interpreter's goto was itself non-conformant. It resolved labels with a forward-only find_label scan of the current instruction list, so:

goto shape old interpreter
forward, same block ✅
backward (manual loop) ❌ raised "goto target not found"
continue (out of an if) ❌ raised
break-style (out of a loop) ❌ raised

So there was no correct oracle to mirror. This PR makes goto/label correct on both engines.

Approach

A tail-recursive list walker has already consumed the block head at a goto, so labels are resolved ahead of execution rather than scanned at runtime. Both engines have analogous cont nesting (:test pushes one entry, loops two), so a goto resolves to {depth-to-unwind, target, close-level} on each — differing only in the target representation.

  • Interpreter (Lua.Compiler.GotoResolution, runs after codegen): rewrites {:goto, name} → {:goto, id} and records proto.goto_targets[id] = {depth, level, target_tail}. The goto handler drops depth cont entries, closes upvalues >= level, and resumes target_tail.
  • Dispatcher (Bytecode.resolve_gotos/2, over the encoded tuple tree): :label → {@op_label, …} no-op; :goto → {@op_goto, depth, target_pc, level}. The handler closes upvalues, unwinds depth cont entries (unwind_goto/2, reusing :break's marker shape), and resumes the destination tuple at target_pc.
  • Close level: scope resolution records each label's live-local register watermark on {:label, name, level}; a goto closes open upvalue cells >= level, matching Lua 5.3 §3.3.4 block-exit (verified via the upvalue-capture test).

do blocks are flat-inlined, so reused label names collapse into one list; resolution picks the nearest label after the goto (else before), matching lexical intent for the continue/break/backward-loop idioms.

Coverage status (honest)

goto/label are now covered. One gap remains that the original exploration missed: short-circuit and/or (:test_and / :test_or) still falls back to the interpreter. It's now the single documented exception in the fully_compiled?/1 corpus guard, left for a focused follow-up. So this PR closes the known gap but the dispatcher is not yet at literal 100%.

Tests / verification

  • New test/lua/vm/goto_test.exs — runs each program on both engines (bytecode stripped vs kept) and asserts equal, correct results: forward/backward/continue/break/nested/reused-label/upvalue-capture.
  • dispatcher_test.exs — a :goto / :label block asserting @op_goto is encoded (helper now descends into nested body tuples) and runs correctly.
  • bytecode_test.exs — goto programs added to the coverage corpus; fallback/cascade tests reworked around and/or.
  • Full suite green: 2443 passed, lua53 suite 17 passed. Dispatcher correctness check: fib(20) dispatcher == interpreter. (Benchee not installed in this env, so the perf comparison wasn't run; the hot path gains only three jump-table clauses for a rare opcode.)

goto.lua skip

goto.lua lines 163-195 (the debug.upvalueid cell-identity assertions) are now skipped: debug.upvalueid is a stub returning nil, so those assertions can't pass once the (now-correct) goto control flow reaches them. The control flow itself stays covered by the preceding assert(#a == 6).

The list interpreter resolved `goto` with a forward-only `find_label`
scan of the current instruction list. Backward jumps (manual loops),
`continue`-style jumps out of an `if`, and break-style jumps out of a
loop all raised "goto target not found"; reused label names could match
the wrong label. A tail-recursive walker has already consumed the block
head at the goto, so labels cannot be resolved at run time.

Resolve them ahead of execution instead. `Lua.Compiler.GotoResolution`
rewrites each `{:goto, name}` to `{:goto, id}` and records
`proto.goto_targets[id] = {depth, level, target_tail}`:

  * `depth` — `cont` entries to drop to leave the blocks between the
    goto and its target (`:test` = 1, loop body = 2), so the remaining
    `cont` is exactly the destination block's entry continuation.
  * `level` — the target label's live-local register watermark (recorded
    by scope resolution on `{:label, name, level}`); the goto closes open
    upvalue cells at or above it, matching Lua 5.3 §3.3.4 block-exit.
  * `target_tail` — the destination block's suffix after the label.

The pass runs after bytecode encoding, so the encoder still reads the
raw goto/label stream (the dispatcher port is separate). New conformance
tests in goto_test.exs cover backward/continue/break/nested/reused-label
and upvalue-capture cases on both engines.

goto.lua: skip the `debug.upvalueid` upvalue-cell-identity assertions
(163-195) — `debug.upvalueid` is a stub returning nil. The goto control
flow those closures exercise stays covered by `assert(#a == 6)`.
With the interpreter's goto now conformant, port goto/label to the
dispatcher so goto programs stop falling back. The encoder resolves
gotos over the encoded tuple tree (mirroring GotoResolution for the
interpreter, with PCs instead of list tails):

  * `:label` → `{@op_label, name, level}`, a no-op anchoring a target PC.
  * `:goto` → `{@op_goto, depth, target_pc, level}` via `resolve_gotos/2`:
    `depth` `cont` entries to drop (`:test` = 1, loop body = 2), the
    label's PC in the destination tuple, and the upvalue-close level.

The dispatcher gains `@op_label` (no-op) and `@op_goto` handlers: close
open upvalues at/above `level`, drop `depth` `cont` entries, and resume
the destination tuple (recovered from the unwound markers via
`unwind_goto/2`, reusing the shape `:break`'s `find_loop_exit/1` relies
on) at `target_pc`. The hot path is untouched — three new jump-table
clauses for a rare opcode.

`Bytecode.fully_compiled?/1`'s corpus now includes goto programs (they
compile end-to-end); the one remaining documented exception is
short-circuit `and`/`or` (`:test_and` / `:test_or`), which the original
exploration had overlooked and which is left for a follow-up. The
dispatcher_test `encodes_op?` helper now descends into nested body
tuples so opcodes emitted inside a block are detected.
@davydog187

Copy link
Copy Markdown
Contributor Author

🤖 Automated multi-agent review — stabilization cycle

5 reviewers fanned out across correctness/conformance/dispatcher/tests/quality dimensions; every finding was then adversarially verified by an independent agent that ran repros against the branch tip and PUC-Lua. Raised 15 → 11 confirmed (3 HIGH, 8 MEDIUM), 4 refuted.

The 11 confirmed findings collapse into 4 distinct issues. The first directly contradicts the PR's central "conformant goto/label" claim, so flagging it before merge.


🔴 HIGH — Flat-inlined do blocks break goto label visibility (§3.3.4), on both engines

Codegen flat-inlines do … end (gen_statement for Statement.Do appends body ++ [close_upvalues] with no wrapper tuple), so a ::label:: inside a do block becomes a plain {:label, name, level} in the parent's flat list. Both resolvers — GotoResolution.find_label_pos/3 (interpreter) and Bytecode.find_label/3 (dispatcher) — then apply a scope-blind "nearest label after the goto, else nearest before" heuristic. Per §3.3.4 a goto may only see labels in its own or an enclosing block, never a nested or sibling one. The moduledoc's claim that "forward-then-backward matches lexical intent" is the precise wrong assumption.

Three independently verified repros (run on branch tip, both engines, vs PUC-Lua):

-- 1) jump INTO a nested block's label
local x=0; goto l; do ::l:: x=100 end; ::l:: x=x+5; return x
-- PUC-Lua: 5   |  this PR: 105  (both engines)

-- 2) backward goto silently binds to a FORWARD sibling-block label
local out=0
do ::a:: out=out+1 if out<2 then goto a end end
do goto a out=out+100 ::a:: out=out+1000 end
return out
-- PUC-Lua: 1002  |  this PR: 1001  (both engines — block-1's loop never runs)

-- 3) sibling-block reuse
local a,b=0,0
do ::l:: a=a+1 if a<2 then goto l end end
do ::l:: b=b+1 if b<3 then goto l end end
return a,b
-- PUC-Lua: 2, 3  |  this PR: 1, 3  (both engines)

Silent wrong results (skipped loop iterations), no error. Fix direction: preserve do-block boundaries through codegen (a scope marker / per-label scope id) so the resolver restricts candidates to the goto's own/enclosing real scope before applying nearest-match. The positional heuristic cannot distinguish a sibling/nested block once flattened. Bytecode.resolve_gotos needs the same fix.

Note: test/lua/vm/goto_test.exs:108 ("reused label name") only exercises the all-forward variant, so it passes by luck — neither backward-vs-forward-sibling collision nor jump-into-nested-block is covered.


🟠 MEDIUM — No static goto/label legality validation (§3.3.4)

PUC-Lua rejects these at compile time; this impl silently accepts and runs them (both engines). Nothing in the parser, Scope.resolve, or GotoResolution validates legality (scope.ex only records the label's register watermark).

-- jump into the scope of a local — reads uninitialised/stale register
do goto skip; local y = 10; ::skip:: return y end   -- PUC: compile error | PR: [nil]
-- with register reuse this can surface an unrelated live value, not just nil

-- duplicate label in the same scope
::a:: ::a:: return 1                                  -- PUC: "label 'a' already defined" | PR: [1]

Fix direction: a label-legality pass (in Scope.resolve or dedicated) that rejects jump-into-local-scope and duplicate labels with PUC-shaped messages, running before both resolvers.


🟠 MEDIUM — Undefined/unreachable goto crashes the compiler with an internal error, and engines diverge

Bytecode.resolve_gotos → resolve_entry/4 does raise "unresolved goto target … — codegen should not emit this". But codegen does emit it: the parser accepts gotos to undefined/out-of-scope labels, so this is reachable from ordinary source.

goto nowhere; return 1
-- PUC: "no visible label 'nowhere' for <goto>" (clean compile error)
-- PR (bytecode path): RuntimeError "unresolved goto target \"nowhere\" — codegen should not emit this" at COMPILE time
-- PR (interpreter fallback, e.g. with an and/or present): "goto target not found" at RUN time

The two passes disagree on the same illegal input (compile-time bare raise vs. runtime InternalError), and GotoResolution.resolve_instr's :error branch + comment ("parser should have rejected this") is effectively dead code on the default path. Fix direction: one shared legality pass returning a proper {:error, CompileError} with a PUC-shaped message for both engines; drop the false "codegen should not emit this" invariant.


🟠 MEDIUM — goto.lua suite skip is mis-categorized (masks the gap above)

test/lua53_skips.exs skips goto.lua lines 12–40 with reason "load() parse-error messages do not match PUC format" (category :stdlib). But those lines use errmsg(...) which first requires load() to fail — they test illegal-goto rejection, not message wording. Since this impl doesn't reject (repeated-label / jump-into-block / jump-over-local all load & run), the :stdlib "message format" reason hides a real §3.3.4 conformance gap from suite-status tracking. Fix: re-categorize as :unimplemented with an honest reason (+ tracking issue), or implement the checks.


4 findings considered and refuted (with reasoning)
  • "PR-claimed rejections have no test coverage" — the PR body claims no such rejections; it scopes itself to valid goto shapes. The static-validation gap is real but pre-existing/out of scope (captured above as MEDIUMs).
  • "Corpus guard only asserts compilability" — by design; execution correctness for every shape (incl. continue/break) is value-checked in goto_test.exs via assert_both/2.
  • "Executor comment documents a 2-tuple, code stores 3-tuple" — real but cosmetic comment drift at executor.ex:657; the authoritative GotoResolution moduledoc is correct; zero functional impact.
  • "goto.lua skip masks untested upvalue-cell freshness" — the close-level feature is value-tested on the forward-exit path; only the distinct-cells-across-backward-passes assertion is missing (minor coverage nicety).

Bottom line: the valid goto shapes the PR targets work and are well-tested, but the "conformant" framing is too strong — the do-block flattening (🔴) produces silently wrong results on legal programs, and static legality (🟠) is unenforced. Recommend addressing the HIGH (or narrowing the PR's conformance claim + tracking the rest) before merge.

I can implement any of these fixes if you'd like — say the word.

Generated by an automated multi-agent review+verify workflow. Findings were verified against branch tip and PUC-Lua but may still contain errors; treat as review signal, not ground truth.

….3.4)

Codegen flat-inlines `do ... end` blocks into the parent instruction list,
so a do-block's labels leaked into the parent's flat label search. Both goto
resolvers (interpreter `GotoResolution`, dispatcher `Bytecode.resolve_gotos`)
used a scope-blind "nearest after, else nearest before" heuristic over that
flattened list, binding a goto to labels in nested or sibling blocks.

Scope resolution now tags every `{:label, name, level, block_path}` and
`{:goto, name, block_path}` with the lexical do-block path (innermost-last,
unique per block). A label is visible to a goto only when the label's path is
a prefix of the goto's — its own block or an enclosing one, never a nested or
sibling block. Both engines apply the same prefix test, so they agree.

Adds `Lua.Compiler.GotoValidation`, a static legality pass that runs before
both resolvers and returns a compile error (surfaced as `Lua.CompilerException`
via `compile/2`/`compile!`) for the programs PUC-Lua rejects at compile time:
duplicate label in a scope, undefined/out-of-scope goto, and a forward jump
into the scope of a local. This removes the dispatcher's bare "codegen should
not emit this" RuntimeError on user-reachable undefined gotos and makes the
two engines diverge no more.

The suite's `load` helper now returns `nil, message` on a compile error (not
raise), matching PUC, so `goto.lua`'s illegal-goto `errmsg(...)` rejections
pass; un-skips goto.lua lines 12-40 (only the debug.upvalueid range remains).

Repros (both engines), PUC value | before -> after:
  local x=0; goto l; do ::l:: x=100 end; ::l:: x=x+5; return x   5    | 105  -> 5
  do ::a:: ... goto a end  do goto a ... ::a:: ... end           1002 | 1001 -> 1002
  do ::l:: ... end  do ::l:: ... end (sibling reuse)             2,3  | 1,3  -> 2,3
`Lua.Parser.parse/1` and `Lua.Compiler.compile/2` both return a binary
error message, so the non-binary `format_load_error/1` clause was dead
(`pattern_match_cov`). Inline the binary message into `load`'s second
return value and remove the helper.
@davydog187

Copy link
Copy Markdown
Contributor Author

✅ Conformance fixes landed (commits 82281ab, 009b880)

The HIGH + MEDIUM findings from the review above are fixed and independently verified against PUC-Lua 5.5 on both engines (not just self-reported green):

Approach: lexical block-path tracking — Scope threads a per-do-block id path; codegen emits {:label, name, level, block_path} / {:goto, name, block_path}; both resolvers (GotoResolution interpreter, Bytecode.resolve_gotos dispatcher) filter candidates by a visible?/2 prefix test (a label is visible iff its block path is a prefix of the goto's — own/enclosing block, never nested or sibling). A new Lua.Compiler.GotoValidation AST pass runs first in Compiler.compile and rejects illegal gotos with PUC-shaped messages.

Verified (interpreter == dispatcher == PUC):

repro before after / PUC
jump into nested do label 105 5
backward goto vs forward sibling label 1001 1002
sibling-block reuse [1,3] [2,3]

Rejections now compile-error (both engines), matching PUC wording:

  • goto nowhere → no visible label 'nowhere' for goto
  • ::a:: ::a:: → label 'a' already defined
  • do goto skip; local y=10; ::skip:: end → jumps into the scope of local 'y'

This also removes the bare "codegen should not emit this" internal raise (BUG 3) — undefined/out-of-scope gotos are now a clean compile error consistently across both engines — and the mis-categorized goto.lua suite skip (BUG 4): the illegal-goto rejection block (lines 12–40) now runs and passes; only the debug.upvalueid range stays skipped.

Verification: mix test 2450 passed · mix test --only lua53 17 passed (goto.lua's errmsg block now exercised) · mix format clean · mix compile --warnings-as-errors clean · Dialyzer clean · full CI green.

Fixes implemented by an automated agent, then independently verified against PUC-Lua (lua-5.5/luac-5.5) and CI by the orchestrator. The first push tripped CI Dialyzer (a dead format_load_error clause); 009b880 fixes it.

@davydog187
davydog187 merged commit 3ce38cc into main Jun 15, 2026
5 checks passed
@davydog187
davydog187 deleted the vm/dispatcher-goto-label branch June 15, 2026 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant